test: Add 46 tests for RAG coverage accuracy improvements - #3

Adds BAAI/bge-base-en-v1.5 model support with automatic query prefix
("Represent this sentence: ") for BGE models. Defaults remain on
all-MiniLM-L6-v2 (384-dim) for production safety; BGE activation
requires re-embedding and Qdrant collection recreation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Activates BM25 hybrid search (enable_bm25_search=True) for keyword
matching alongside vector similarity. Upgrades reranker to
BAAI/bge-reranker-base (110M params). Reverts embedding defaults to
all-MiniLM-L6-v2 (384-dim) to match production .env configuration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Passes full transcript/document text to ContextualEnricher for better
chunk context. Uses DeepSeek cache (system+transcript static per video,
chunk varies) for cost efficiency. Adds enrichment_version column to
chunks (v1=original, v2=contextual) via migration 018.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Adds LLM-based relevance grading that runs after reranking to detect
REFORMULATE/EXPAND_SCOPE/INSUFFICIENT scenarios. Disabled by default
(enable_relevance_grading=False) - zero production impact until enabled.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Generates hypothetical answer passages for coverage queries to improve
recall. Uses max-score fusion to merge HyDE results with primary search.
Disabled by default (enable_hyde=False) - enable after BGE migration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Updates test mocks to match current codebase:
- Add documents_used/limit/remaining to QuotaUsage constructors
- Update fact extraction prompt threshold and keyword assertions
- Fix conversation history tests: add is_query kwarg to embed_text mocks,
  use LLMResponse instead of SimpleNamespace, add flush/users to FakeSession,
  disable query expansion/rewriting in unit tests, update chunk resolution
  test for new chunk_id requirement

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…st coverage

Phase 2-4 of Two-Level Retrieval plan:
- Theme aggregation: frequency-based topic extraction from collection videos
- Video similarity: Jaccard similarity on key_topics with shared topic display
- LLM clustering: k-means on summary embeddings + LLM-generated theme labels
- Wire up TwoLevelRetriever to replace inline retrieval in conversations.py

New endpoints:
- GET /collections/{id}/themes (cached aggregation)
- GET /collections/{id}/themes/clustered (stored clusters)
- POST /collections/{id}/themes/regenerate (async Celery task)
- GET /videos/{id}/similar (Jaccard-based similarity)

Frontend: CollectionThemes and SimilarVideos components integrated.

Test coverage improvements:
- video_tasks.py: 11% → 69% (25 tests)
- enrichment.py: 21% → 98% (30 tests)
- vector_store.py: 34% → 86% (32 tests)
- theme_service.py: 90% (23 tests)
- video_similarity: 19 tests
- theme_clustering: 19 tests

Total: 545 tests passing, 0 regressions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Next.js requires useSearchParams() to be inside a Suspense boundary
for static generation. Fixes build error on /conversations route.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Suggested starter questions in empty conversations (context-aware for single vs multi-source)
- Conversation search and sort (by recent, message count, alphabetical)
- Conversation export to markdown via actions menu
- Last message preview in conversations list (backend subquery + frontend display)
- One-click "Chat with this" button on collection cards
- Copy message to clipboard with visual feedback on assistant responses

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Batch Qdrant upserts to 500 points per request (fixes 32MB payload limit)
- Increase Celery task timeout to 2 hours (was 60 min, large docs need ~78 min)
- Cache document extraction to skip re-extraction on reprocess
- Clean up old chunks and vectors before reprocessing (prevents duplicates)
- Always clear error_message on status transitions (fixes stale error display)
- Use PDF metadata page_count when available (fixes incorrect page counts)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add usePaginationParams + PaginationBar to conversations, documents,
collections, admin users, admin conversations, and admin Q&A feed.
Replace custom pagination JSX on admin pages with shared components.
Make PaginationBar itemLabel configurable (default "items").
Update getCollections API to accept skip/limit params.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Replace static 4-card layout with interactive split-panel design:
- Clickable step tabs (left) with animated visual previews (right)
- Auto-advance every 5s with progress bar, pause on click, resume after 8s idle
- 4 animated preview components using framer-motion (URL input, transcript, indexing, citations)
- Updated messaging: "From Video to Verified Answer" with outcome-focused step titles
- Mobile: stacked cards with inline visual previews
- Technical credibility footer line

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Cover intent classifier pattern collisions, cross-source boundary
behavior, two-level retriever coverage fallback pipeline, chunk
limit edge cases, prefetch scaling, and backfill task/endpoint.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Introduce a contract-based approach to catch behavioral bugs proactively.
Defines 14 contracts (MEM-*, CIT-*, ACC-*, PAR-*, RET-*) covering memory
retention, citation accuracy, storage calculations, and content parity.

New proactive skills:
- conversation-quality: triggers on memory/fact files, detects dead zones
- citation-accuracy: triggers on citation files, validates tracking
- content-parity: triggers on task files, checks doc/video parity

New manual skill:
- behavioral-contracts: full audit of all contracts before releases

17 contract unit tests (15 passing, 3 skipped documenting known issues).
Updates rag-architect, rag-best-practices, rag-quality-gate, and
test-before-complete with behavioral contract verification.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>